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The Alabama Superintendent's Report Card: 

An Analysis of Local Education Agency Characteristics 
and Success at Meeting State Defined Performance Measures 

Introduction 

State defined performance indicators for education are rapidly becoming the 
hallmark of the 1990s. As of 1996, 52 of 54 state education agencies had at least one 
annual accountability or indicator report (Council of Chief State School Officers, 
1996). Thirty-two states now have some form of accountability legislation enacted 
that require the reporting of school and school system performance indicators. Most 
of the analysis and literature on the development of state-defined performance 
indicators describe a pattern of implementation with little prior conceptual 
development. Inter-school or school district ratings and comparisons are central to 
much of this legislation and focus, most often, on student performance measures 
and school finance data. These type of indicators provide more information on 
school and school system need than they do on the quality of education that is 
provided. The challenge in developing accountability legislation is to define 
assessment strategies and performance indicators that support reforming school 
practice and assure access to educational outcomes. Are these measures to be used 
for reinvention and renewal, or intervention? 

Student and school indicators have become additional points of focus in 
accountability efforts and legislative mandates. The Council of Chief State 
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School Officers (CCSSO) State Education Policies Reports have attempted to provide 
state-by-state information on policies relating to time of attendance, content 
standards, high school graduation requirements, and teacher licensure. In addition, 
the CCSSO has assisted states in developing performance indicators at the district 
and school levels. Research on issues related to state-mandated accountability 
efforts is increasing. A major challenge for researchers is defining a context for the 
use of indicators that supports the planning and management of schools toward the 
end of increased teaching and learning. Odden (1990) argues that indicators by 
themselves cannot provide policy relevant information in support of strengthening 
educational processes unless used within the context of a strong conceptual 
framework. One such framework, defined in terms of educational inputs, 
educational processes and educational outcomes, was put forth by Shavelson, 
McDonnel & Oakes, (1989). In this major study for the Rand Corporation the 
authors concluded that performance indicators which support strengthening 
teaching and learning, and the publics right to information on the effectiveness of 
schools, meet the following eight criteria. 

They reflect core features of the educational system. 

They measure factors that policy can influence. 

They provide information to support decision making on current or potential 
problems. 

They measure observed behavior rather than perceptions. 

Use valid and reliable measures. 




4 



Mathews /Alabama Report Card 



4 



• Use data that can be collected. 

• Provide analytical linkages among indicators. 

• Address a broad range of audiences. 

Section 16-6B-7 of the Education Accountability plan for the State of Alabama, 
adopted July 7, 1995, outlines that the State Board of Education (Alabama Code, 1995) 
will report annually to the public for each school and area. The legislation calls in 
general terms for reporting in three areas: a funding and expenditures report; a 
student achievement report ; and a school safety and discipline report. This 
legislation has been translated by the State Education Agency (SEA) into the 
Alabama State Superintendent's Report Card (1996). This document provides data 
on 14 performance indicators that define some measure of accountability for the 128 
local education agencies (LEA) in the Alabama system. The report card was 
published for the first time following the 1995-96 academic year. 

In its first iteration, the SEA attempted to use report card to describe and 
characterize school district financial, enrollment, and teacher certification level and 
to link these with the student achievement of each school district in the state. The 
school district report card contained 14 indicators in the following categories: (a) 
Stanford Achievement Tests, (b) system OLSAT, (c) ACT test scores, (d) high school 
exit exam scores, (e) writing test scores, (0 enrollment, (g) attendance, (h) per pupil 
expenditure in ADA, (i) free and reduced lunch eligibility, and (j) professional 
certification, (k) state, local, and federal revenues, and (1) mills equivalent. 
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Included in the Alabama State Superintendent's Report Card for each LEA 
was a letter grade score for select measures. For the 1996 Report Card, there was no 
conceptual framework that defined the quality of the LEA's efforts to accomplish 
teaching and learning, given the attributes of the student population. The current 
study was conducted to determine if a conceptual framework for school district 
performance could be developed using the information provided in the Alabama 
LEA level report card. For purposes of analysis, 17 additional indicator variables 
were taken from the State Superintendent's Statistical and Financial Data and added 
to the Report Card indicators. The additional indicators include (a) 14 financial 
indicators, (b) dropouts, and (c) percentage of district enrollment by race. 

The Problem 

Interpretation of the information in the Report Card by the public is an 
important consideration in publishing and disseminating the document statewide. 
Is the information in the 1995-96 Alabama Report Card sufficient to inform the 
public, legislature and media about the achievement of each school and school 
district as was intended by the Education Accountability Plan (Alabama Code, 1995). 
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Purpose of the Study 

The purpose of this study was to analyze the indicators in the 1996 Alabama 
Report Card (State of Alabama State Superintendent's Report Card, 1996) and 
determine the relationship between these indicators, student achievement, and 
grade ratings for all 127 school districts. Other indicators were obtained from the 
State Superintendents's Annual Statistical and Financial Data (1995). 

The Research Questions 

The basic research question addresses the need for information in the Report 
Card to "inform the public about student achievement in each school" as was 
intended of the Education Accountability Plan for accountability reports to the 
public (Alabama Code, 1995). The research questions for this study are: (a) What are 
the descriptive profiles of Alabama school districts related to student achievement? 
and (2) What is the difference in computed school district grade definition levels 
based on demographic indicator variables and the achievement levels assigned to 
school districts by the Alabama State Department of Education? 

The Alabama Report Card 

A school district Report Card has two categories of indicators and contains a 
total of 14 indicators. The first category is labeled achievement indicators with the 
following: (a) Stanford Achievement Tests, (b) system OLSAT, and (c) ACT test 
scores. The second category is labeled other indicators and lists the following: (a) 
high school exit exam scores, (b) writing test scores, (c) enrollment, (d) attendance, 
(e) per pupil expenditure in ADA, (f) free and reduced lunch eligibility, (g) 
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professional certification, (h) state, local, and federal revenues, and (i) mills 
equivalent. For purposes of analysis, 17 additional indicator variables were taken 
from the State Superintendent's Statistical and Financial Data (Alabama State 
Department of Education, 1995) and added to the Report Card indicators. The 
additional indicators include (a) 14 financial indicators, (b) dropouts, and (c) 
percentage of district enrollment by race. 



Student Achievement Assessment Scheme for the 1996 Report Card 

Grading Definition Assigned to School Districts Representing Overall Achievement 



A Superior 

A- Excellent 
B+ Outstanding 
B Very Good 

B- Good 

C+ High Average 
C Average 

C- Moderate Caution 
D+ Caution 

D Caution 

D- Extreme Caution 
F Failing 



Percentile rank of 70 or above 

66 to 69 

62-65 

58-61 

54-57 

48-53 

40-47 

35-39 

31-34 

27-30 

23-26 

22 and below 
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Mills Equivalent 
(Local Effort) 

Good Above 40 mills 

Fair 30 to 40 mills 

Poor Below 30 mills 

Stanford Achievement Test Grade Definition 
(SAT 9 Percentile Ranking Scores) 



A 


70 and above 


C 


40 to 47 


A- 


66 to 69 


C- 


35 to 39 


B + 


62 to 65 


D+ 


31 to 34 


B 


58 to 61 


D 


27 to 30 


B- 


54 to 57 


D- 


23 to 26 


C+ 


48 to 53 


F 


22 and below 



5TH- AND 7TH-Grade Writing Test 
Level of Proficiency 

Level IV: Superior Performance beyond Level III Course or Grade level mastery. 

Level 111: Solid Academic Performance at Course or Grade Level 

Level 11: Partial Mastery of Knowledge and Skills Fundamental for Work at 

Course or Grade Level 

Level I: Minimal Grasp of Knowledge and Skills Fundamental for Work at 

Course or Grade Level 

High School Basic Skills Exit Exam Grade Definition : (Percent Passing Reading, 
Language, and Math) 



A 


96 to 100 


B+ 


93 to 95 


B 


90 to 92 


C+ 


87 to 89 


c 


83 to 86 


c- 


79 to 82 


D 


75 to 78 


D- 


70 to 74 


F 


Below 70 
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Operational Definitions 

gradgJP ffmitip n ; Letter grades, A through F, assigned by the Alabama State 
Department of Education indicating the achievement level of school districts based 
on SAT (9) achievement test scores scaled by percentile rank. 

Average Daily Attendance (ADA): The aggregate attendance of a school during a 
reporting period divided by the number of days school is in session during this 
period. Only days on which the pupils are under the guidance and direction of 
teachers are considered as days in session. The average daily attendance for groups of 
schools having varying lengths of terms is the sum of the average daily attendance 
obtained for the individual schools. State average in ADA is 96. 1%. 

Current Expenditures Per Child in ADA: The expenditures for operating local public 
schools, excluding capital outlay and interest on school debt. These expenditures 
include such items as salaries for school personnel, fixed charges, student 
transportation, school books 

and materials, and energy costs. These items are divided by the ADA. State average 
is 54.458. Southern Regional Education Board stalest average is 55.220. 

Stanford Achievement Test (SAT 9) Scores: Nationally normed achievement test 
comparing performance in reading, mathematics, language, science and social 
science to other students in the nation: national average is the 50th percentile. 

Otis-Lennon School Ability Test (OLSAT): A test measuring school ability (verbal 

and quantitative) compared to other students in the 

nation in the same grade; national average is the 50th percentile. 

High School Basic Skills Exit Exam: The percentage of 1 11th grade students who 
passed each section of the Exit Exam on their first attempt. The state average is 86% 
for reading. 82% ~ for mathematics, 78% for language. 

American College Testing (ACT): This is the average ACT test score for students 
who took the test. The national average is 20.8. The state average is 20.0. 

School Safety & Discipline: Standards will be developed for the 1996-97 school year. 
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Mills Equivalent: The total amount of revenue collected locally, minus insurance 
and tuition, for public school purposes divided by the value of one regular district 
mill of ad valorem tax. The state average is 36.1 mills equivalent. 

Percent State Revenue: The percent of revenues from the state sources divided by 
the total state, local and federal revenues. The national average is 46%. The state 
average is 69.3%. 

Percent Local Revenue: The percent of revenues from local sources divided by the 
total state, local and federal revenues. The national average is 46.7%. The state 
average is 19.3%. 

Percent Federal Revenue: The percent of revenues from federal sources divided by 
the total state. Local and federal revenues. The national average is 7.2%. The state 
average is 11.3%. 

Enrollment: The number of students on attendance roll as of the 40th day of school 
opening. 

Professional Certificates: Issued based on the completion of approved programs and 
other requirements at the following levels: Class B (bachelor's degree level). Class A 
(master's degree level) and Class AA (sixth-year level). 

6th- and 7th-Grade Writing Test: A direct assessment of writing which determines 
the level of proficiency. The highest level being Level IV and lowest level being 
Level 1. The assigned proficiency level is based on the level where the largest 
portion of the students scored. 

State Revenue in ADA : Revenue from state sources in dollars per student average 
daily attendance. 

Local Revenue in ADA : Revenue from local sources in dollars per student average 
daily attendance. 

Federal Revenue in ADA: Revenue from federal sources in dollars per student 
average daily attendance. 

Instructional Expenditure in ADA: Instructional expenditures per student in 
average daily attendance. 

Percent of Students Eligible for Free/reduced Lunch: Percent of student enrollment 
eligible under federal guidelines for free or reduced lunch. 
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Percent Non-white Student Enrollment: Percent of students enrolled in the school 
district who are listed as minority race. 

Percent White Student Enrollment: Percent of students enrolled in the school 
district who are listed as white race. 



Methods 

Research Desig n 

A causal comparative research design was utilized. The population in this 
study consisted of all 127 public school districts in Alabama. Since the entire 
population of school districts in Alabama was used in the study, the results of the 
study are applicable only to school districts in Alabama for that particular school 
year. Data for this study on school district indicator variables and grade definition 
levels were obtained from school district Alabama Report Cards (1996). Report Card 
data were obtained from the Alabama State Department of Education and was in 
electronic format on 3.5 inch computer diskettes. Additional hard copy data were 
collected from the Superintendent's Annual Statistical and Financial Report (1995). 
Analysis 

For Research Question 1, means and frequency distributions were computed 
to provide a descriptive profile of school districts in the 1996 Alabama Report Card. 
For Research Question 2, discriminant analysis was employed to determine the 
computed school district student achievement grade definition using school district 
variables that described the school districts within the context of their demographic 
profile. The discriminant analysis compared the computed levels of grade definition 
using demographic profile indicators with the student achievement grade definition 
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assigned by the Alabama State Department of Education based on SAT (9) 
standardized test scores. The results of these analyses are discussed in the findings of 
this study. 

Findings 

Descriptive Profile 

This section contains a descriptive profile of indicators summarizing each 
indicator variable (See Table 1, Appendix A). The indicator variables presented in 
this section are: (a) SAT9 grade definition, (b) state revenue in ADA, (c) local 
revenue in ADA, (d) federal revenue in ADA, (e) expenditure per pupil in ADA , (f) 
instructional expenditure in ADA, (g) local district mills of revenue, (h) percent of 
students eligible for free/ reduced lunch, (I) percent non-white student enrollment, 
and (j) percent white student enrollment. 

Summary of Descriptive Analysis 
Based on Demographic Indicators 

The mean values of the state revenue in ADA, federal revenue in ADA, 
expenditure per pupil, percent eligible for free/reduced lunch, and percent of non- 
white student enrollment indicators decreased as school district grade definition 
increased. The mean values of the local revenue in ADA, local district mills of 
revenue, and percent of white student enrollment increased as school district grade 
definition increased. Instructional expenditures appeared to be more equally 
distributed throughout the grade distribution range (See Table 1, Appendix A). 
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These descriptive statistics indicate that as the poverty of an LEA increases, as 
indicated by eligibility for a free or reduced lunch, the lower the grade on the 
Alabama State Superintendents Report Card. The same relationship holds for 
percentage of non-white students. The descriptive statistics imply a link between 
poverty, race and the State Superintendents measure for poorly performing school 
districts. As would be expected, these school districts are receiving a greater share of 
the state equalization funds and federal funds. Expenditure data shows evidence 
that the more poorly performing LEA's expend more per pupil, most likely with 
federal funds. However, instructional expenditures per ADA are fairly constant 
across all school districts . The only real anomaly was seen with the 10 highest 
scoring school districts. These LEA's spent more per ADA pupil and for 
instructional ADA than all the other 118 districts. 

It seems clear that the State of Alabama State Superintendent's Report Card, 
1996 demonstrates that there are differences in the populations being served by the 
various school districts. It does not stand as an instrument for public reporting on 
the quality of teaching and learning. Nor does it speak to policy at any but the most 
macro level. 

Discriminant Analysis of Computed and Assigned Grade Definition 
Levels Based on Demographic Indicator Variables 

The indicator variables presented in this section are: (a) grade definition, (b) 
state revenue in ADA, (c) local revenue in ADA, (d) federal revenue in ADA, (e) 
expenditure per pupil in ADA , (f) instructional expenditure in ADA, (g) local 
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district mills of revenue, (h) percent of students eligible for free/reduced lunch, (I) 
percent non-white student enrollment, and (j) percent white student enrollment. 
The indicators included in this procedure were to determine if any differences 
existed between computed school district student achievement grade definition and 
the student achievement grade definition assigned by the Alabama State 
Department of Education based on Report Card indicators and indicators from the 
Superintendent's Annual Statistical and Financial Report (1995). The dependent or 
grouping variable was school district grade definition. The structure coefficients 
shown in Table 2 indicate the relative importance of the independent variables in 
four discriminant functions calculated in the equation. When viewing the structure 
matrix, independent variables ordered by size of the correlation between the 
variables and the discriminant scores define the discriminant functions. 

Four indicators met the minimum Wilks' Lambda criteria as discriminating 
variables that defined the discriminant functions (see Table 3). These four indicators 
were: (a) percent students eligible for free/reduced lunch, (b) percent white student 
enrollment, (c) federal revenues, and (c) instructional expenditures. 
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Table 2 

Stepwise Discriminant Analysis: Structure Coefficients Between Independent Variables and Canonical Discriminant 
Functions 









Discriminant Functions 




Variable 


1 


2 


3 


4 


Percent students eligible 


for free lunch 




.040 


.209 


.401 


Percent non-white 


student enrollment 


-.741“ 


-.527 


-.395 


.055 


Percent white 


student enrollment 


736“ 


.536 


-.410 


•051 


Local revenues 


182 


-.519“ 


-.244 


•303 


Mills equivalent 


056 


-.363“ 


-.150 


.014 


Federal revenues 


-.610 


-.403 


.642“ 


.228 


Instructional expenditures 


in ada 


092 


-.492 


-.283 


.818“ 


Expenditures per pupil 


in ada 


059 


-.494 


-.186 


.741“ 


State revenues 


-.101 


-.005 


.223 


.606“ 



Note. Variables are ordered by the size of the correlation within each function. 

“Denotes largest absolute correlation between each variable and any discriminant function. 



16 

o 

ERIC 



Mathews /Alabama Report Card 



Table 3 

Summary of Stepwise Discriminant Analysis for Levels of Grade Definition: Wilks' Lambda of 
Independent Variables 





Variables in the equation 




Step 


Variable entered 


Wilks' Lambda 


prob. 


1 


Percent students 








eligible for free/reduced 
lunch 


.215 


.000* 


2 


Percent white student 
enrollment 


.154 


.000* 


3 


Federal revenues 


.117 


.000* 


4 


Instructional expense 


.098 


.000* 



*p < .05 



The results of the discriminant analysis, which employed the demographic 
indicator variables, indicated that approximately 53% (n=53) of the school districts 
were classified by discriminant analysis in grade definition levels different from 
grade definition levels assigned by the State Department of Education (see Table 4). 
Using the demographic indicator variables, the discriminant analysis classified 
approximately 47% of the school districts in the same levels as those assigned by the 
State Department of Education. 
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Summary of Analysis of Demographic Indicators 
Using Discriminant Analysis 

Discriminant analysis was used to determine which demographic indicators 
were important predictor variables used to classify school districts into one of 11 
grade definition levels. The discriminating variables that were defined by the 
discriminant analysis to classify school districts within one of 11 accreditation levels 
were (a) percent students eligible for free/reduced lunch, (b) percent white student 
enrollment, (c) federal revenues, and (c) instructional expenditures. Based on the 
discriminant analysis using the seven demographic indicator variables, 53 (53%) of 
the 127 school districts were classified in accreditation levels different from those 
assigned by the State Department of Education. 

Conclusions and Recommendations 

Conclusions 

What are the descriptive profiles of Alabama school districts related to 
student achievement? The descriptive analysis in this study presented a clear, 
comparable picture of the relationship between grade definition levels and the 
indicators profiling the demographic context of school districts. Even though school 
districts assigned to caution levels of grade definition generated less local revenues 
than the districts assigned to good, outstanding, and excellent levels, state revenues 
appeared to have equalized the disparities in locally generated funds. The higher 
levels of expenditures per pupil in the lower grade definition districts can possibly 
be explained by the higher levels of federal revenues and higher percent of students 
eligible for free/ reduced lunches. The 10 school districts assigned to the superior 
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level of grade definition spent approximately 400 dollars more per pupil on 
instruction than districts assigned to the lower levels of grade definition. However, 
there appears to be very little difference in the per pupil instructional expenditures 
for school districts assigned D through A-. 

What is the difference in computed school district grade definition levels 
based on demographic indicator variables and the achievement levels assigned to 
school districts by the Alabama State Department of Education? It can be concluded 
that the Report Card, in its present format, does not discriminate well in assigning 
school districts to grade definition levels D through A on the systematic 
examination of the nine demographic profile indicator variables in this study. 

The link between school district grade definition levels and the 1996 Alabama 
Report Card indicator variables appears tenuous when comparing the use of the 
SAT9 test scores versus the demographic profile indicators . Given the tenuous link 
between the grade definition levels and the Report Card the SAT9 scores versus the 
demographic indicator variables, the accreditation system and the Report Card 
format needs some rethinking or reexamination regarding the policy for assigning 
grade definitions to school districts. 

Recommendations 

The findings indicated that the use of the additional demographic indicator 
variables, not included in the Report Card, provided valuable information about 
school district grade definition status. These additional indicators more clearly 
described the relationship between the demographic profile of school districts and 
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grade definition; and, they were significant predictors of grade definition levels in 
the discriminant analysis. It can be recommended that further studies be conducted 
to determine how additional demographic indicator variables impact school district 
grade definition assignment. 

Further studies should be conducted to determine why school districts in 
grade levels D through C- are not spending an equivalent per pupil expenditure on 
instruction compared to school districts in level A. Qualitative case studies of 
particular school districts are needed to determine how and why these school 
districts in grade definition levels D through C- are not successful and why school 
districts in grade definition levels C through A are successful. These are the why and 
how questions this study and the 1996 Alabama Report Card did not answer. 

The 1996-97 Alabama Report Card; The Future for Accountability Reports 

While the 1996 Alabama Report Card used only 14 variables to report the 
condition of education at the school district level, the 1997 report card (not yet 
published) will include school level data as well as district level data. The new 1997 
report card will far exceed the 1996 report card with the information needed for 
analysis and policy decisions to improve education in Alabama schools. The 
Alabama State Department of Education (SDE) 1997 report card was developed 
through collaboration of numerous educators and community members around the 
state. In an effort to address low achievement, the SDE , through the 1995 
Educational Accountability Plan, provides assistance programs for school districts 
with at-risk students. A minimum of $100 per student has been allocated to provide 
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tutorial assistance programs, after-school, Saturday school, and/or summer school 
to assist identified at-risk students, schools, and school districts in raising levels of 
achievement. 
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Appendix A 



Table 1. Means and Ns of Indicators by Grade Definition (SPSS Output) 
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